The Engel algorithm for absorbing Markov 

chains 



J. Laurie Snell * 

Version dated circa 1979 
GNU FDLt 



Abstract 

In this module, suitable for use in an introductory probability 
course, we present Engel's chip-moving algorithm for finding the basic 
descriptive quantities for an absorbing Markov chain, and prove that 
it works. The tricky part of the proof involves showing that the initial 
distribution of chips recurs. At the time of writing (circa 1979) no 
published proof of this was available, though Engel had stated that 
such a proof had been found by L. Scheller. 
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In [1] Arthur Engel provides an interesting algorithm for finding the basic 
descriptive quantities for an absorbing Markov chain. It was developed as a 
teaching tool. It can be carried out by moving chips around the transition 
graph of the Markov chain. In [2] Engel provides examples and a partial 
proof that the algorithm works. We complete his proof here. 
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Let P be the transition matrix of an absorbing Markov chain with states 
1,2, ... ,r transient and states r + 1, . . . , s absorbing. Assume that the tran- 
sition probabihties are rational numbers and write them in the form pij = ^ 
where rj and Vij are integers. Note that if we have chips in state i, we can 
distribute these chips according to P by sending rij to state j. 

We choose a particular transient state u. The algorithm will find N^j, 
the expected number of visits to state j when the process is started in state 
u, and Buj, the probability that the process ends in absorbing state j when 
it is started in state u. We put an initial disbribution of chips ci, C2, . . . , 
on the transient states. The choice q = — 1 for i 7^ m and Cu = Tu is called 
a critical loading. We put no chips on the absorbing states initially. We are 
allowed to make two types of moves. A type 1 move is to move chips from 
state i by moving to state j for each j. A type 2 move is to add a chip to 
our starting state u. A type 1 move is possible only when there are at least 
chips on state i. We also assume that we are only allowed to make a type 2 
move when no type 1 move is possible. We begin with a critical loading and 
then make as many type 1 moves as possible. When no more such moves are 
possible, we employ type 2 moves until we have r„ chips at u. We then see 
if we have again a critical loading. If not, we continue the process. We stop 
when we obtain a critical loading again. We shall prove that this will occur. 

Let Wuj be the total number of chips that have been moved out of the 
transient state j and Vuk the number of chips that have been moved into the 
absorbing state k during the algorithm. Let Vu = J2k'^uk- Then we shall 
show that (a) the algorithm will stop, and (b) N^j = ^ and Buj = 

First we shall illustrate the algorithm by a simple example. We consider 
random walk with a drift with state and 3 absorbing. (See Figure [TJ) 
We choose u to be state 1. The critical loading is (3,2). Carrying out the 
algorithm (see Tabled]), we obtain wu = 9, W12 = 6, fio = 3, ^13 = 4, and 
vi = 7. Thus Nn = 9/7, = 6/7, B^o = 3/7, and ^13 = 4/7. 

Let us carry out these calculations by the standard method. We put P 
in canonical form 
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Figure 1: Random walk with a drift. 
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Then 



N = ii-Qy 





and 





B = NR 



We turn now to the proofs. Let us start with any initial loading less 
than or equal to the critical loading. We continue the algorithm until some 
distribution 6i, 62, • • • , &r is repeated. There must be such a distribution since 
there are only a finite number of distributions that can occur. 

We now start over again, this time with the initial loading 61, 62? • • • ? ^r- 
We shall follow the proof in [2J to show that our foumulas for N^j and B^j 
are correct. We shall use only the fact that the initial distribution repeats. 
We shall then prove that the critical loading will always repeat. 

Since the initial and final distributions are the same, the number of chips 
that move into a state must equal the number that move out. Thus 
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Table 1: A run of Engel's algorithm 
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= Yl '^ukPki for i 7^ 
Let = — . Then we have 

k 

Wui = XI for i 7^ 

If we do this for all choices of u, we obtain the matrix equations 

W^I + WQ 

or 

W{I-Q) = I 

and 

w = {i- qy^ = N. 

Similarly for j absorbing 

"^uj ^ ! ^ukPkj- 
k 
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Putting Vuj = — , we obtain 

k 

Doing this for all u, we obtain in matrix form 

V = WR. 

But since W = N, we have 

V = NR = B. 

We show now that the critical loading will repeat. In [2] it is stated that 
L. Scheller has proved this fact but the proof is not given. The following 
proof was provided by Peter Doyle. 

We know that there is an initial sequence bi,b2, ■ ■ ■ ,br that repeats. We 
can assume that hi < ri ioi i ^ u and 6„ = r„. We choose this sequence 
using green chips. We then add red chips to bring this distribution up to the 
critical loading. We now carry out the same sequence of moves using only 
the green chips. Then when our initial distribution repeats, we will have the 
critical loading again since we have not touched the red chips. Let us call 
this modified procedure method A. We call it 'modified' because we have not 
followed the Engel rules for our initial load — the critical loading. Assume 
that in method A we added m chips to state u. Then we consider a second 
method which we call method B. In method B, we start with the critical 
loading and carry out the Engel rules until we have added m chips and can 
make no more type 1 moves. We will show that the two methods lead to the 
same final distribution. Since we know that the final distribution in method 
A is the critical loading, the same will be true for method B. 

Let ei, 62, . . . , e„ be the moves in order as made in method A and /i, /2, . . . , 
those made in method B. Moves Ci and /i are the same, namely move r„ 
chips from u. We shall prove by induction that all the e moves appear some- 
where in the / list. Assume that we have, for the moves ei, 62, . . . , e^, found 
corresponding moves fi-^, fi^, . . . , fi^. among the /'s. Consider move e^+i. If 
this is a type 2 move, there must be a corresponding move /j^.^^ since we 
make the same number m of such moves in these two methods. Assume then 
that Cfc+i is a type 1 move, say move rj chips from state t. If there is such 
a move /j^.^^ among the /'s which occurs in addition to fi-^, fi^, . . . , fi^ and 
before these are all carried out, we can choose this move to correspond to 
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Cfe+i. Assume that there is no such move. The move e^+i was made possible 
by the moves Ci, 62, . . . , e^. This move must also be possible by virtue of the 
moves /jj , /jj , . . . , fi^, . The other moves that could have occurred while these 
moves are made can only help by adding chips to state t. Thus again, we 
can match Ck+i and the induction step is proved. 

The same argument show that all the /^'s are among the e^'s. Thus the 
two sets of moves are the same, and this implies that methods A and B lead 
to the same final distribution, as we wanted to show. 
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